Sparsity-Aware Orthogonal Initialization of Deep Neural Networks

Deep neural networks have achieved impressive pattern recognition and generative abilities on complex tasks by developing larger, deeper models, but these are increasingly costly to train implement. There is in tandem interest develop sparse versions of powerful models post-processing with weight pruning or dynamic training. However, it requires expensive train-prune-finetune cycles compromises the trainability very deep network configurations. We introduce sparsity-aware orthogonal initialization (SAO), a method initialize maximally connected weights. SAO constructs topology leveraging Ramanujan expander graphs assure connectivity assigns weights attain approximate dynamical isometry. Network sparsity tunable prior model compared fully-connected demonstrated that outperform magnitude up thousand layers fewer computations training iterations. Convolutional special constraints, while kernel may be interpreted as tuning level. Within framework, kernels pruned based desired compression factor rather than post-training parameter-dependent heuristics. well-suited for applications tight energy computation budgets such edge computing tasks, because achieves sparse, trainable learnable parameters without requiring layers, additional training, scaling, regularization. The advantages attributed both its initialization.